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The purpose of this paper is to obtain Plancherel formula for Berezin kernel 
representations. We obtain an explicit expression for density of the Plancherel 
measure for all 10 series of classical Lie groups for sufficiently large values of 
parameter of representation (precisely for the case when representation is square 
integrable). A statement of the problem is contained in our Subsection 1.4 (see 
also recent works on this subject @, §, @, 0, ||, |l4j, @). A solution 
is contained in §9. 

An evaluation of the Plancherel measure is reduced to an evaluation of some 
integrals over symmetric spaces. The main matter of the paper is this evaluation. 
In fact we evaluate integrals, which are more general than it is necessary for the 
Plancherel formula. Nevertheless neither calculations, nor final formulas doesn't 
become more complicated 

We give explicit Plancherel formula only for the groups 0(p, q). For all other 
series the Plancherel formula can be easily obtained in the same way by the a 
combination of integrals (O.l)-(O.ll), (0.14)-(0.15), (0.17)-(0.18) with classical 
Gindikin - Karpelevich formula. 

I thanks V.F.Molchanov for discussion of this subject. 

The structure of the paper is following. Section contains a list of 
integrals that we have evaluated. Evaluations themself are contained in §§4-8. 
The simplest cases are discussed in §4. The basic case is contained in Subsection 
5.1. The main obstacle is surmounted in Subsection 6.1. All other Subsections 
of §5-8 contain some technical (but not obvious) variations of calculations of 
5.1 and 6.1. 

In § § 1—3 we discuss different models of Riemann noncompact symmetric 
spaces (matrix balls, matrics cones, matrix wedges, and sections of wedges). 
This information have no relation to calculations themself. We only explain, 
what domains of integration in §§4-8 are symmetric spaces. 

In §9 we obtaine the Plancherel formula itself (formula 9.2). 

§0. Formula for Integrals 

0.0. Notations. By K we denote R, C, or the algebra of quaternions 
H. Denote the quaternionic imaginary units by i, j, k. We denote by * the 
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operation of transposition of a matrix. Denote by A* the matrix adjoint to a 
matrix A (i.e. A is the matrix obtained from A by conjugation and (complex 
or quaternionic) conjugation of matrix elements). For real matrices A* := A*. 
The symbol is introduced by formula (1.1). 

By [g] P we denote p x p left upper block of a matrix g. 

The symbol det(g) denote determinant. Let g be an operator in H™. Then 
we consider g as an operator g R : R 4n — > M 4n and assume det(g) = tydet gm- 

The symbol T ^> means that a matrix T is hermitian (i.e. T = T*) positive 
definite matrix (i.e. qTq* > for any row q). 

A matrix R is named dissipative if R + R* 3> 0. For any dissipative matrix 
R and for any complex r we define quantity det i? T = det(i? T ). Let Ai, . . . , A„ 
be eigenvalues of the matrix R. Then real parts of the complex numbers Xj are 
positive. We assume 

det R T := cxp{r In A^} 
In the quaternionic case, we define powers of determinant by the formula 

det R T = deti?R /4 . 

We also use nonstandard notation 

:= a a a\ a,X,reC 

and the similar notation for complex dissipative matrices R: 

deti? {CT||T} := det iTdetif 

The symbol dT, where T is matrix, denote the Lebesgue measure on space 
of matrices. We also fix following normalization of Lesbegue measure. Let for 
instance T pass the space ofnxn hermitian matrices. Denote by tkk diagonal 
matrix elements of the matrix T, by tk m = Pkm + iqkm we denote non-diagonal 
elements. Then we assume 

n 

dT=Y\_dt kk Yi dp km dq km 

k—l l^fc<m^n 

In the same way, we define a normalization for Lebesgue measure for spaces of 
matrices of other types (symmetric, antisymmetric etc.) 

We start the list of integrals over symmetric spaces evaluated in the paper 
(§§4-8). Discussion of models of symmetric spaces is contained below in §§1-3. 

In all cases Xj,aj,Tj £ C, we also assume 

An+i = 0; cr n+ i = 0; r„+i = 

We don't formulate explicitely conditions of convergence since they can be easyly 
observed from explicit formula. 
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0.1. Integrals over spaces GL(n,R)/0(n), GL(n, C)/U(n), GL(n,H)/Sp(n). 
Denote by C n (K) the space of positive defined n x n hcrmitian matrices over 
K. In §4 we establish the following formula: 



™ deHTl A3 ~ A3+1 

yi^pi^ detr-(" +1 )/ 2 rfT= (o.i) 



1)/2 T(A fe - (fc - l)/2)r(<7 fe - A fc - (n - fc)/2) 



(fc-l)/2 



fc=l 



r(a fc - (n - fc)/2) 



" det[T] Aj " Aj+1 

IT , / x det T- n dT = (0.2) 

_ -q ^fc-i r(A fc - k + l)r((j fc - A fc - n + fc) 



fc=l 



r(o-fc - n + fe) 



/■ " det[T] 

/ TT , , r n \ detT-( 2 "- 1 )rfT= (0.3) 



n 

fc=l 



2fc-2 r ( A fe - 2fc + 2)r(cr fe - Afc - 2n + 2fc) 



r(crfc - 2n + 2fc) 



0.2. Integrals over the spaces 0(n, n)/0(n)xO(n), U(n, n)/U(n)xU(n), 
Sp(n, n)/Sp(n) x Sp(n). We denote by W„(K) the space ofnxn dissipative 
matrices R over K. We represent a dissipative matrix R in the form R = T + S 
where T = T* > 0, S* = —S. In §5 we evaluate the following integrals: 



L 



" dctfTl A '~ A3+1 

ff -± — — — det T-"dT dS = (0.4) 

t+s ew„(R) det[l + T + S\^~ 1 



n 



^- x r(Afc - (n + fc)/2 + l)I>fc - Afc - (n - fe)/2) 



fc=i 

n Aj-A, + i 



r((jfc - n + fe) 
/■ " det[T] Aj ~ A '+ 

ir+s Gff„(C) }J[ det[l + T + S] 



1 -± n T det T~ 2n dT dS = (0.5) 



n 2 2(n-fc+l)-«T fc -r fc 2fc-l F ( A fc ~ n ~ fc + 1 ) r K + Tfc - Afc - n + fc) 
r(afc - n + k)T(r k - n + k) 

r " dct[T] A ^ Aj+1 

Jt+s ew n (M) jJl dct[l + T + S] j 

n 2 a fe -4(n-fc+i) 4fc-3 r ( A fc ~ 2(n + fc) + l)T(gfc - A fc - 2(n - k)) 
,.. =1 " r(a„/2 - 2(n - fc))r(o- n /2 - 2(n - A) - 1) 



l —l det T- in dT dS = (0.6) 



3 



0.3. Integrals over the spaces 0(p, q)/0(p) x 0(q), U(p, q)/V(p) x U(g), 
Sp(p, <?)/Sp(p) x Sp(g). Let p < q. Let us define a space SW Piq (K). Its points 
are pairs (K, L), where K is a p x p matrix, L is a p x (q — p) matrix, and 

^(K + K*) - LL* > 

We represent if in the form K = M + N, where M — M*, N = — N*. In §6 
we evaluate the following integrals 

f p dct\M — LL*] Xj ~ Xj+1 

/ TT ^ „ • det(M - LL*)-( p+q V 2 dMdNdL = (0.7) 

Jsw p , qm jJi det[l + M + N}^ +1 

_ JJ T(A fc - (g + AQ/2 + l)L(a fc - A fc - (p - fc)/2) 



T(a k -p + k) 



I 



n 



det[M-LL*]^ 



sw Pl ,(C) jt x i det[l + M + N ]1^-^^-t ]+i} 

x det(M - LL*)~( p+q } dM dN dL = (0.8) 

n 2 2(p-k)-a k -r k7r 2k-l r ( A fc ~ (g + fc) + l)r(cr fc + Tfc - Afc - P + fc) 
. =1 L(a fe -p + fc)L(r fe -p + fc) 



/ TT ^ detfM - LL*)- 2(p+q UMdNdL = (0.9) 



'sif M (i) " det[l + M + N]j 



n 2 4(p-k+l)-a k7v 2( q +k)-3 F ( A fc - 2 (g + fc) + l)r(0-fc - Afc - 2{ P - fc)) 
r(<T fe /2 - 2(p - fc))r(<7 fe /2 - 2(p - fc) - 1) 



fe=i 

(we assume A p +i = cr p +i = t p +i = 0). 

0.4. Integrals over the spaces Sp(2n, R)/U(n) and Sp(2n, <C)/Sp(n). 
We realize the symmetric space Sp(2n, R)/U(n) as the space of all symmetric 
n x n matrices R with a positive definite real part. Let us represent R in the 
form R = T + iS, where matrices T, S are real. In §7 we evaluate the following 
integral 



n 



r " det rr 3 , M , 

/ TT LJi r detT"(" +1 )rfTdS= (0.10) 

2 2- CTfc -r fc +„-fe^fe r(Afe _ (n + k y 2 )T{a k + Tfc - Afc - (n - fc)/2) 



r(<7 fe - (n - fc)/2)r(r fc - (n - fc)/2) 

We realize the space Sp(2n, C)/Sp(n) as the space of pairs (T, 5) of complex 
n x n matrices satisfying the conditions 

T = T* > 0, S = S* 
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Let j be the quaternionic imaginary unit. Then 

!A S -A S+1 



/ S 



det[T\: 



T=T*>0, S=S* 



L = * det[l + T + Sj] 



dct T-^+^dTdS = 



(0.11) 



=n*r 

k=l 



2k _ x T(A fc - (n + k))T{a k - A fc - (n - fc)) 
T{a k - 2{n - k) - 1) 



0.5. Integrals over the spaces 0(n, C)/0(n), SO*(2n)/U(n). Let 

\ 



/0 -1 
1 



V 



-1 

1 



(0.12) 



We realize the space 0(2n, C)/0(2n) (respectively SO*(4n)/U(2n)) as the space 
of 2n x 2n matrices R over R (respectively over C), satisfying the conditions 



J ^ R^ J — R'. 



R + R* > 



(0.13) 



Assume R = T+S, where T = T*,S = -S*. Then, in the case 0(2n, C)/0(2rc), 
we have 



n 



det[T]< Aj ~ Aj+l)/2 



J2j 



. detT -(2n-l)/2 dTd5 



L A ! dct[l + T + S]g^ +l)/2 

2fe-2 r (^fe - (n + fc) + 2)r(c7 fe - A fe - (n - fc)) 



(0.14) 



In the case SO*(4n)/U(2n), we obtain 



det[T]g 3 " Aj+l)/2 



Udetll + T- w;,y 



{(<t 3 - ( t j + i)/2||(t j -t j + i)/2} 



r(a fe - (n - fc)) 



detT -(2«-l) rfTd5 



(0.15) 



n 2 2+4(n ~ 



k=l 



L(A fc - 2(n + fc) + 3)r(o- fc + r fc - A fc - 2(n - fc)) 
r(<7 fc - 2(n - k))T(T k - 2(n - fc)) 



We realize spaces 0(2n + l,C)/0(2n + 1) and SO*(4n + 2)/U(2n + 1) as 
spaces of (1 + 1 + 2ri) x (1 + 1 + 2n) matrices R, having block structure 



'\ * N 
R = | 1 

,0 * *; 



(0.16) 
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and satisfying condition (0.13). Then in the case 0(2?i + l,C)/0(2n + 1) we 
have 



det[i? + i? tl(A ^ A3+l)/2 
] 

_ x T(\ k - (n + k) + l)T(a k - X k - {n - k)) 



TT T7^det(i? + i?*)-"di?= (0.17) 

fJl det[i + i?]g;-^ +l)/2 



fc=i 

and in the case SO*(4n + 2)/U(2n + 1) we have 
det[i? + i?*i (A ^ A:, ' +l)/2 



T(a k - (n - k)) 



/ TT ^7 ^™ Tm dct(fl + R*r 2n dR = (0.18) 



i=1 det[l + i?] 2j+2 

= r TT 4 fc-l r ( A fc - 2 (" + fc) + 2 ) r K + ^ - A fc - 2 (" - fc )) 

where C\, C2 are some powers of 2. 

0.5. Known partial cases, a) Hua Loo Keng integrals ((l0)) are integrals 
over matrix balls (see below §1) having type 



/ 



det(l - zz*) a dz 



They can be reduced to very partial cases of our integrals by means of the 
Cay ley transform z = (1 + i?,) _1 (l — R). 

b) Integrals (0.1)-(0.3) for Ai = • • • = A„ can be reduced by substitution 
T + 1 = (H — 1) _1 to a partial case of Gindikin B-function of cone, see see 
also @, Theorem VII.1.7. 

c) Integrals (0.1)-(0.3) for ci = ■ ■ ■ = <j n were evaluated by Faraut and 
Koranyi (see. §, XIV.4). 

d) Berezin in one of the last his papers (Q) announced the Plancherel formula 
for kernel representations of the groups V(p,q), Sp(2n,R), SO*(2n). In fact, it 
is equivalent to an evalution of integrals (0.5), (0.8), (0.10), (0.15), (0.18) for 
<7i = ti = (T2 = i"2 = . . . (up to meromorphic multiplier depending on er)^ 
Proofs were published by Upmeier and Unterberger |2(J. Our method differs 
from J2(| and (as far as I know) it differs from Berezin method. 

e) Plancherel formula is known for rank 1 classical groups, i.e. 0(n, 1), 
Sp(n, 1), see. 0, This case is essentially more simple than the case of rank ^ 2. 
(our calculations in this case are trivial). 

2 after submitting of the paper I received interesting preprint of G.Zhang | ^]| generalizing 
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§1. Models of Symmetric Spaces: Matrix Balls 



1.1. Matrix balls. By a norm of a matrix we always mean the norm of 
a operator from a coordinate Euclidean space K p to W. By a matrix ball we 
mean 

the set of all p x p matrices over K having a norm < 1 (where K = R, C, H) 

or set of all n x n matrices over K, satisfying one of symmetry condi- 
tions: symmetric (z — z), antisymmetric (z = —z), hermitian (z = z*), 
anti-hermitian {z = —z*). 

Let G be one of classical groups GL(n,R), GL(n,C), GL(n,H), 0(p,q), 
U(p,q), Sp(p,g), Sp(2n,R), Sp(2n,C), 0(n,C), S0*(2n). Let K be a maximal 
compact subgroup in G. Any non-compact classical Riemann symmetric space 
G/K has matrix ball type realization (see 0)). Let us enumerate these real- 
izations. Emphasis that in all cases we consider a space of matrices with norm 
< 1. 

1. 0(p, q)/0(p) x O(q) is the space of p x q matrices over R. 

2. GL(n, R)/0(n) is the space of symmetric matrices n x n matrices over R. 

3. 0(n, C)/0(n) - is the space of antisymmetric n x n matrices over R. 

4. U(p, q)/\J{p) x U(q) is the space of p x g matrices over C. 

5. Sp(2n, R)/U(n) is the space of symmetric n x n matrices over C 

6. SO*(2n)/U(n) is the space of antisymmetric n x n matrices over C 

7. GL(n, C)/U(n) is the space of hermitian n x n matrices over C 

8. Sp(p, g)/Sp(p) x Sp(g) is the space px? matrices over H. 

9. GL(n,Hf)/Sp(n) is the space of hermitian nxn matrices H 

10. Sp(2n, C)/Sp(n) is the space ofnxn anti-hermitian matrices over H 

In all cases, the group G acts on the matrix ball by fractional linear trans- 
formations having the form 



z 



[g] 



(a + zj)~ 1 ((3 + zS) (1.1) 



Moreover, the group G consist of all fractional linear transformations, mapping 
bijectively matrix ball to itself. Let us describe the group G explicitly. 

In the cases 1,4,8 (i.e. in the case G = ~U(p,q,K)) we consider the space 

W © K 9 equipped with indefinite hermitian form M = \ \ ^ J . We realize 



v -1 f 

the pseudo-unitary group U(p, q,K) over K (i.e. one of the groups 0(p,q), 
U(p, q), Sp(p, q)) as the group of block matrices g — ( " ^ 
(j) + q) x (p + q) and preserving the hermitian form M. 
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In all other cases, we consider the space K n © 1" equipped with indefinite 
hermitian form M = ( ^ ] . As above elements of the group G preserve 



s -1 , 

this form and also they preserve once more form A depending on type of the 
matrix space: 

a) in the case of a space of symmetric matrices ( G/K = GL(n,R)/0(n), 
Sp(2n, R)/U(n)), the form A is an antisymmetric bilinear form with the matrix 

1 
-1 

b) in the case of a space of antisymmetric matrices {G/H = 0(n, C)/0(n, R), 
SO*(2n)/U(n)), the form A is a symmetric bilinear form with the matrix ^ ^ ^ 

c) in the case of a space of hermitian matrices ( G/K = GL(n,R)/0(n), 
GL(n, C) /U(n), GL(n, H)/Sp(n)), the form A is an anti-hermitian form with the 

matrix ( ^ ) ' rem ^ n< ^ that a sesquilinear form A is named anti-hermitian 



if A(«, w) = —A(w, v) for all v, w. 

d) in the case of a space of anti-hermitian matrices ( G/H = Sp(2n, C)/Sp(n)), 

the form A is a hermitian form with the matrix 



1 / 

In all cases, a stabilizer of the point z — consists of matrices f ^ ^ 

G, where matrices a and 8 are unitary over K; note also that in all cases, exept 
G = U(p,g,K)/U(p,K) x U(q,K), we have 6 = a*~\ 

1.2. Jacobian. Lemma 1.1. A differential of a map z t— ► in a point z 

is 

(a + zjy 1 dz(-'yz [9] + 6) 
Proof. We have to calculate the following expression up to o(e) 

{a + (z + eu)7) _1 (& + (z + eu)5) = 

= (a + 27) _1 (1 + euj(a + z-fy 1 )' 1 (f3 + zS + euS) = 
= (a + zjy x (l - euj(a + z^y 1 + o(s))([3 + z5 + euS) = 
= (a + z7) -1 (/3 + z8) + e(a + z^^u [-j(a + 27) _1 (/3 + zS) + 5]+ o(e) = 

= z [g] + e(a + z7)~ 1 m(-7z [! ' 1 + S) + o(e) ■ 

Recall a formula for the determinant of block matrix (see Q,§11.5). Below 
it is used very intensively 

det ( C D I = det A ' det ^ ~ CA ~ lB ) ( L2 ) 
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Lemma 1.2. 

dct(- 7 z[ 9 l + (5) = (dct g) det(a + Z7)- 1 (1.3) 

Proof. 

dct(- 7 z [9] + (5) = det(-7(a + z 7 )" 1 (/3 + z5) + S) 
By formula (1.2), we reduce this expression to the form 



det(a + , 7 )-det( a+ 7 ^ = 



= det(a + z7) 1 dct ^ " ^ 
Lemma 1.1, 1.2 impliy the following statement 

Lemma 1.3. A Jacobian of a transformation g 1— ► in a point z is 

|det(a + zi)\~ h 

where h = (p + q) dimK in the cases 1,4,8, and h = 2ndimG/K in other cases. 

1.3. Invariant measure. 

Lemma 1.4. 

1 - z^(z^)* = (a + 27) -1 (l - zz*)(a + zj)*- 1 (1.4) 

Proof. 

1 - = 1 - (a + z 7 )- 1 (/3 + z«5)(/F + S*z*)(a* + 7* z*)" 1 = 

= (a + *7) _1 {( a + + 7*^*) - (0 + z5){f3* + S*z*)}(a* + 7* z*)' 1 (1.5) 

By 

a /3 \ / 1 W« (?V /l 
7 * J V -lj^ 7 «5 J ~ ^ -1 

we obtain that the factor in (1.5) in the curly brackets has required form. ■ 
Corollary 1.5. 

dct(l-zM(zM)*) = |dct(a + z 7 )|- 2 det(l-zz*) 

Proposition 1.6. A G-invariant measure on a matrix ball is given by the 
formula 

|det(l - zz*)\- h ' 2 dz 

where h is the same as in Lemma 1.3, and dz denote Lebesgue measure on the 
space of matrices. 
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Proof. It is consequence of Lemma 1.3 and Corollary 1.5. 

1.4. Kernel representations. Consider a function on the matrix ball 
G/K given by the formula 

= det(l - zz*Y a 
We can consider the function 23 as a function on the the group G 

■B a (g) :=23 Q (0 [9] ) 

Kernel representation T a of the group G is the unitary representation containing 
a cyclic vector v such that 

(T a (g)v,v) =-B a (g) 



Representation T a exists iff the function 23 a (g) on the group G is positive 
definite. Conditions of positive definiteness for 23 Q (g) were obtained by Berezin 
in || (some additions were given by Gindikin For instance, Berezin condi- 
tion for the groups 0(p, q) is 

2a = 0, 1,2, . . . ,p - 1 or 2a>p-l 

(we assume p ^ q). For other series conditions are similar. 

Kernel representation also can be defined as restrictions of highest weight 
representation of some 'overgroup' G* D G to G. They were introduced by 
Berezin for hermitian symmetric spaces, in 16 , jl3| there were observed that 



the problem is general for all Riemannian noncompact symmetric spaces (besides 
several exceptional spaces). Detail discussion and more 'material' definitions see 
ID, ||, see also recent works @, 0], @, @. 



§2. Models of Symmetric Spaces: Symmetric Cones and 
Symmetric Wedges 

Matrix ball models are not convenient for our calculations. Our purpose 
in §2-§3 is to obtain realizations at which parabolic subgroups acts by affine 
transformations. Our realizations are similar to realizations from JTg|] . 

2.1. Cayley transform. The Cayley transform is the transform of the 
space of square matrices given by the formula 

l + il + R)- 1 (2.1) 
-l + tl + z)- 1 (2.2) 



R = 



1 - R 

1 + R ' 

1 - z 



1 + JB 
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By Lemma 1.1, the differential of the Cayley transform is given by the for- 
mula 

dR = 2{l + z)- 1 dz(l + z)- 1 

Proposition 2.1. a) zz* = 1 R = -R* 

b) ||*|| <1 <^=> i? + i?*>0 

c) z = z l R = R f 

d) z = z* R = R* 

(Statements a),b) are well-known, and c),d) are obvious.) 

2.2. Symmetric cones. Proposition 2.1b),d) implies 
Corollary 2.2. The Cayley transform maps the matrix ball 

GL(n,K)/U(n,K) = 

= GL(n,R)/0(n), GL(n,C)/U(n), GL(n,H)/Sp(n) 

to the cone of positive definite hermitian matrices over K = R, C, H respectively. 

Thus we obtaine another realization for symmetric spaces enumerated in 
the Corollary. The group G = GL(n, K) acts on the cone of positive definite 
matrices by the formula 

A:T^ ATA*. 



2.2. Wedge of dissipative matrices. Proposition 2.1b) also implies 
Corollary 2.3. The Cayley transform maps matrix ball 

U(n,n,K)/U(n,K) x U(n,K) = 

= 0(n,n)/0(n)xO(n), U(n,n)/U(n)xU(n), Sp(n,n)/Sp(n) x Sp(n) 

io <fte space o/ a/? dissipative matrices over K — R, C, H respectively. 

In this case, it is convenient to realize the group G = U(n, n, IK) as the group 

of matrices ^ ^ ^ ^ preserving the hermitian form ^ ^ ) ' 

The group G acts on the space of dissipative matrices by fractional linear 
transformations 

R^ RW : = ( a + Rc)- 1 (b + Rd) (2.3) 



2.4. Siegel upper half-plane. Recall that the matrix ball Sp(2n, R)/U(n) 
consists of complex symmetric matrices. By Proposition 2.1 b),c) its image 
under the Cayley transform is the space of complex symmetric matrices with 
positive definite real part. 

2.5. The space Sp(2n, C)/Sp(n). In this case the matrix ball consists of 
quaternionic anti-hermitian matrices. Consider the modified Cayley transform 

R = -l + 2(l + iz)- 1 
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The image of this transform consist of quaternionic matrices R satisfying 
the conditions 



R* 



i^Ri; i? + i?*>0. 



(2.4) 



Represent a matrix R in the form R = S + Tj, where R, S are complex matrices. 
Condition (2.4) is equivalent to 



5' = 5' t ; T>0. 



2.6. Spaces 0(2n, C)/0(2n, R), SO*(4n)/U(2n). Recall that these matrix 
balls consist of 2n x 2n antisymmetric matrices over R, C respectively. Define 
the matrix 



J = 



/0 -1 
1 



V 



-1 

1 



\ 
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(2.5) 



Consider the modified Cayley transform 

J2 = — 1 + (1 + JZ)- 1 
The image of the matrix ball consists of matrices R satisfying the conditions 
R t = JRJ~ 1 ; i? + i?*>0 (2.6) 

We will continue a discussion of these spaces in §8. 

2.7. Invariant measure. Lemma 2.4. Let z and R are linked by the 
Cayley transform (2.1). Then 



l-zz* = 2(1 + R)- 1 ^ + R*)(l + R*) 



Proof. 1 - zz* = 1 - (1 + i?) _1 (l - R)(l - R*){1 + R*)' 1 = 

= (1 + R)- 1 {(1 + R)(l + R*) - (1 - R)(l - R*)} (1 + R*)- 1 

and we remove brackets in the middle factor. ■ 
The Lemma implies two corollaries. In the thirst place, it allows to rewrite 
functions T> a (see Subsection 1.4) in new coordinates. In all cases (see Subsec- 
tions 2.1-2.6) we obtain 



det(2(R + R*)) 
|dct(l + i?)| 2 



(2.7) 
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In the second place, Lemma 2.4 and Proposition 1.6 give formula for G-invariant 
measure on a cone or a wedge. In all enumerated cases the invariant measure is 
given by the formula 

dct(R + R*) dl " l< ™ K) dR (2.8) 

where m is the size of matrices, which we consider. 

2.8. Action of a parabolic subgroup. In the case of symmetric cones 
GL(n, K)/U(n, K), we consider the minimal parabolic subgroup f C GL(n,K) 
consisting of lower-triangular matrices. Obviously, functions 

n 

]Jdet[T]^~ A3+1 (2.9) 

i=i 

are eigenfunctions of the subgroup P. 

Consider the case when a space G/K is a wedge of dissipative matrices (i.e 
G/K — U(n, n, K)/U(n, K) x U(n, K)), Let us define the group V consisting of 
all transformations 

R i ^ QRQ* + S 

of the wedge, where Q € GL(n, K) is a lower-triangular matrix and S = —S*. 
It is easy to see, that J 1 is a minimal parabolic subgroup in G = U(n, n, K). 

Eigenfunctions of the parabolic subgroup V on the wedge have the form (2.9) 
where T = (R + R*)/2. 

For all other wedges the picture is similar. 

2.9. Comments to exterior of integrals §0. Now we are ready to 
explain a sense of some factors in integrands (0.1)-(0.18). The last factor is 
density of the invariant measure, see (2.8), it is possible to include this factor to 
numerator of the fraction). The numerator of the fraction YTj=i det[T]^ J Aj+1 
is an eigenfunction of the minimal parabolic subgroup. 

I don't understand the sense of the product in denominator. In any case (see 
(2. 7), (2. 8)) all expressions of the form "B a {R)-Y\]=i det[T] A3 ~ Aj+1 are contained 
in the set of our integrands. 

§3. Models of Symmetric Spaces: Sections of Wedges 

In this section we discuss symmetric spaces G/K which have no matrix cone 
and matrix wedge realizations (it is the case when the group G is not split). 

3.1. The case G = V(p 7 q)/V(p) x XJ(q). Denote by B Qi/3 = B a>/3 (K) the 
space of all a x f3 matrices over K with norm < 1. Recal that we identifies 
B Q;j g(K) with symmetric space \J(a,(3, K)/U(a,K) x U(/3, K), see Subsection 
1.1. 
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Assume p < q. Let us wright Z G B p ^ as block p x ((q — p)) + p) matrix: 

Z = ( X Y ) 

Complete the matrix Z to the matrix Z of the size ((q — p))+p) x ((<? — p)) + p) 
by the formula 





X Y 



(3.1) 



The Cayley transform (2.1) maps this matrix Z to a dissipative matrix i? having 
the form 

where 

if = -1 + 2(1 + y) _1 ; L = -(1 + Y)- 1 X 

Conversely, it is easy to check, that any dissipative matrix having form (3.2) is 
a Cayley transform of some matrix having form (3.1). 

Thus we realized the space V(p,q, K) /U(p, K) x U(g, K) as the space SW p . q — 
SW p , g (K) of dissipative matrices having block structure (3.2). 

Let us represent the matrix (3.2) in the form 

R = ( 2L M + n) (3 ' 3) 
where M = M* > 0, N = —N*. Then 

Formula (1.2) implies the identities 
det 



1 L* 
L M 



det[M-LL*]j (3.5) 



q-p+j 
L M , 



det ( l L * ) = det(M - LL*) (3.6) 



Lemma 3.1. Dissipativity of the matrix (3.3) is equivalent to positive defi- 
niteness of the matrix M — LL* . 

Proof. It is enough to apply Sylvester criterion, see (3.5). ■ 

3.2. Minimal parabolic subgroup. 

Lemma 3.2. The group of all fractional linear maps of the wedge W q map- 
ping the set SWp^ q to itself is isomorphic to the group V(q — p, K) x V(p, q, K). 
Moreover elements of the first factor U(q — p) fix all points of SW Piq . 
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PROOF. The statement is equivalent to the following fact. The group H 
of all fractional linear transformations of the matrix ball, B q . q mapping the set 
of matrices (3.1) to itself, is U(q — p, K) x U(p, q, K). First the inclusion H D 
U(g — p, K) x U(p, q, K) is obvious. Secondly the group U(g — p, K) x U(p, q, K) 
is a maximal subgroup in U(g, q). ■ 

Let us consider the group U(g, g, K) acting on the wedge W q and maximal 
parabolic subgroup Q C U(, ) consisting of transformations 

R^URU*+H (3.7) 

where U € GL(n,K), a, H = —H*. Consider the group 

S = Qn(U(g-p)xU(p,g,K)) 

which consist of all transformations (3.7) mapping the set SW P:q to itself. It is 
easy show that these transformations have the form 



1 0\ [A OW 1 Q\fA* C*\ ( -AC* 



,2L if/ VC D)\2L Kj\0 D*J 1 ^CM* Z 

1 

2{DLA* +CA*) DKD* +CC* + 2DLC 



(3.8) 



where AA* = 1; Z = -Z* 

By construction, § C \J(q — p, K) x U(p, g, K). But the factor U(g — p, K) fix 
all points of the set SW p , q Hence we can assume § C U(p, q, K). 

Formula (3.8) seems cumbersome. Let us give more pleasant description. 
The transformation group § is generated by the following transformations 

1. transformations (i, K) i— » (LA* , if) forming the group U(g — p, K), 

2. transformations (L,K) (DL, DKD*) forming the group GL(p, K), 

3. transformations (i, if) (L, K + Z) forming an Abel group. 

4. transformations (L, K) ^ (L + C,K + CC* + 2LC*). 

The transformations of the type 4 don't form group. Nevertheless the trans- 
formations of types 3 and 4 generate a metabelian group of high 2. 

It is readily seen, that § is a maximal parabolic subgroup in \J(p, q, K). We 
extract a minimal parabolic subgroup IP from § by the condition: the matrix D 
is lower-triangular. 

Obviously, cigenfunctions of the minimal parabolic subgroup § on the space 



SWp^ q have the form 



n 



det 

l 



1 L* 
L M 



det[M - LL*]j j Aj+1 



q-p+j 

These functions are restrictions to SW q of Q-eigenfunctions on W q 
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3.3. Functions 23 Q in our coordinates on SW Pt q have the form 

_ f det(2(R + R*)) \ a _ ( det(4(M-LL*) \ a 
a[n> ~ \ | dct(l + i?)| 2 J \\dct(l + M + N)\*J 1 ' 

(they also are restrictions to SW q of functions T> a on W q ). 

3.4. The spaces 0(2ra + l,C)/0(2n + 1), SO*(4n + 2)/U(2n + 1). In 
Subsection 1.1 these spaces were realized as spaces of anticymmetric (2n + 
1) x (2n + 1) matrices over R, C with norm < 1. Let us embed the space of 
(2n + 1) x (2n + l)-matrices to the space of (2n + 2) x (2n + 2)-matrices. For 
this purpose we wright zero column on the left and zeros row above. Then we 
apply the construction of Subsection 2.6 and obtain the models described above 
in Subsection 0.5. 



§4. Calculations in Matrix Cones. 

In these section we evaluate integrals (0.1)-(0.2) over symmetric cones GL(n, 
GL(n, C)/U(n), GL(n, H)/Sp(n). In generally, these 3 series are essentially more 
simple than the others, see for instance Our calculations in these case also 
are essentially simplified. 

4.1. Separation of variables in the cone of real symmetric matrices. 

Thus we have the integral 

tt det^^_ (n+1)/ 

Represent a matrix T as block (1 + (n — 1)) x (1 + (n — 1)) matrix: 

P q* 



T = 

By formula (1.2), 



q r 



detT^detP-ir-qP^q 1 ) (4.2) 

det(l +T) = dct(l + P)(l +r-q(l + P) _1 g*) (4.3) 

In new coordinates, the domain of integration T 3> is replaced to 

P > 0; r - qP^q 1 > 

(it is consequence of Sylvester criterion and equality (4.2)). Using these nota- 
tions we wright the original integral in the form 

"- 2 det[P]^- Aj+1 detpA^x-Cn+D/2 
Al det(l + [Pjj)^-^ ' det(l + P)' 7 "- 1 X 

x {r ~ qP- 1 q t ) Xn ' {n+1)/2 {l + r - q{\ + P)- 1 q t y' In dqdr dP 
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Let us change variable u = r — qP q t (we change r to u, other variables 
are the same ). We obtain 



n-2 j„ + rol A j- A 



n 



det[P]j S 3+1 detP \ n -i-(n+i)/2 



X 



p»o \f = \ det (! + [P]^-^ det(l + pyn-i 

X I M A„-(n+l)/2 | x + u + q p-l q t _ q (J + P)- 1 ^! an dqdu)dP 

/u>0,geB" _1 ^ ' J 

Next we reduce the expression in the curly brackets to the form 

{l + u + g(P(l + P))~Y} 
Then we change the variable q — (qi, . . . , q n -i) G M n_1 to the variable 

h = {hi, . . . , hn-i) = q(P(l + P)T 1/2 G M"- 1 , 
leaving the variables P, u the same. The Jacobian of the substitution is 

dctP 1 / 2 det(l + P) 1 / 2 . (4.4) 
Thus integral can be decomposed to a product 

t A, — A, 



;3>>0 det (i + [p] j)CTJ -^ +1 • det(1 + py n _ 1 -x/2 dP x ( 4 - 5 ) 



det[P]; 3_A3+1 detP A "-^»/ 2 
let(l + P) ff »-i- x 

u ^-C«+i)/2{! + u + hh t }~" n dhdu (4.6) 

The first fuctor (4.5) has a form (4.1) with other parameters (the parameter n 
is replaced to n — 1, and also a are changed). Hence we can apply to integral 
(4.5) the same arguments. It remains to calculate the second factor. 

4.2. Several lemmas. Now we will give without proof several standard 
propositions which will be intensively used below. 

Lemma 4.1. The area of an unit sphere in M. k is 

s =rm (47) 

Lemma 4.2. (see. 3.1.2.1) 

f(u + v)u> J '- l v v - l dudv = B(ii,v) J f{w)w^ +l/ - 1 dw (4.8) 

u>0»>0 m>0 
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Lemma 4.3. 



/ /(! + * + y)-^y^ dx dy = r(b)r(c) r r ( ( a a ) " b ~ C) (4.9) 

ir>0 j/>0 

(We substitute z = u,w = ii + tito the integral (4.8) and integrate in z; 
integral (4.9) can be easily reduced to (4.8)). 

4.3. Calculation of separated factor. The factor (4.6) is a partial case 
of the following integral. 

Lemma 4.4. 

[ u^-\l + u + \h\r°du dh = ^ X - - A > (4.10) 

PROOF. Passing to spherical coordinates (see (4.7)) in the variable h — 
(hi, ... , h n -i), we obtain 



f f u x - k ' 2 -\l + u + p 2 )-° P k - l dpdu 

Ju>0 J p>0 



27r fc / 2 

r(fc/2) . 

_fe/2 



~ r(*/2) 

The last expression is a partial case of integral (4.9). ■ 

4.4. Complex and quaternionic cases are very similar to real case . 
Formula (4.4) for a Jacobian is replaced to 

det p dimK / 2 det(l + P) dimK / 2 

Then all is reduced to Lemma 4.4. 

§5. Integrals over Wedges of Dissipative Matrices. 



In Subsection 2.3 we realized the spaces 0(n, u)/0(n)xO(n), U(n, n)/U(ra)x 
U(n), Sp(n, n)/Sp(n) x Sp(n) as spaces of dissipative matrices. Let us represent 
a matrix R in the form R = T + S, where T = T* , S = -S*. 

5.1. Evaluation for the spaces 0(n,n)/0(ra) x 0(n). Consider the 
integral 

/ T — „ „ dot T- n dTdS (5.1) 

T=T*»0S=-S' J_i J 
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Let us represent the matrices T, S as block ((n — 1) + 1) x ((n — 1) + 1) 
matrices: 

H p < q :y. s-(t -:) 

By formula (1.2), we obtain 

detT = detP- (r-qP- 1 q t ); 

det(l + T + S 1 ) = det(l + P + A) • (1 + r - (q + 6)(1 + P + A)~ 1 (q t - 6')) 

(5.2) 

Substituting u = r — qP~ 1 q t , we obtain 

f dPdA(f\ dct ^" A,+1 detP^- x 

J VfJf det[l + P + A]* i_<Tj ' +1 det(l + P + A)^-i 

Ju^-^ll + u + qP- 1 ^ - (q + b)(l + P + A)- 1 (q t -b t )y Jn dudqdb 



P^>0,A=A t 

X 



u>0,q,b£R" 



Let us wright the expression in the curly brackets as 



(5.3) 



where 



+ q b )x( £ )} (5.4) 



, P- 1 -(1 + P + A)- 1 (1 + P + A)- 1 
' =: I -(1 + P + A)- 1 (1 + P + A)- 1 ) 



Unfortunately the matrix X is not symmetric, hence we wright (5.4) in the 
form 

I , . w„ . / (J* 



1 + u + - ( q b)(X + X t ) 



Lemma 5.1. a) The matrix ^(X + X f ) is positive definite. 
b) det(i(X + X*)) = dctP- 1 dct(l + P + ^)- 2 
Proof, b) \{X + X r ) = 

P- 1 - 1(1 + P + A)- 1 - 1(1 + P - A)- 1 1(1 + P + .4)- 1 - 1(1 + P - A)- 1 
1(1 + P- A)- 1 - ^(l + P + A)- 1 i(i + p + J 4)-i + + A)- 1 

Adding the second column to the first row, and the second row to the first 
row, we obtain 
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det(i(X + {1 + p_ A) -i i (1 + p + {1 A yi + ff 1 + p _ A) -i ) 

Applying formula (1.2), we obtain that the original determinant equals to 
detP" 1 x 

xdctQ(l + P + yl)- 1 + i(l + P-A)- 1 -(l + P-A)- 1 P(l + P + A)- 1 ^ 
= dct P- 1 det(l + P + A)- 1 det(l + P - A)" 1 x 

xdetQ(l + P-A) + i(l + P + A) 

It remains to notice, that the expression in the big brackets equals 1 and (1 + 
P + A) f = 1 + P — A. This complete our evaluation. 

a) The expression det(l + T + S) doesn't vanish. By formula (5.2), u and 

(5.4) also don't vanish. Thus the quadratic summand ( q 6 ) A ^ ^ t ^ is non- 
negative. Since the evaluated determinant doesn't vanish, we obtain required 
statement. ■ 

Then we change the variables q e E" _1 , b e R n_1 to the variable h e 
in integral (5.3) by the formula 



h=(q b)yJ±(X + X*) (5.6) 



All other variables remain the same. The Jacobian of the substitution is det P 1 / 2 det(l+ 
P + A) . This yields that our integral decomposes to a product of two integrals 



n-2 j„j.n->l-\) _ \i 



ndet[P]7 + 1 dt pX n - 1 -n+l/2 
— „ a • , /? „ 7~\ T dPdAx (5.7) 
■ 1 det[l + P + A]' 7 /-° i+1 det 1 + P + A 



/ 

J U 



u^- n {l + u + \h\ 2 )- a "dudh (5.8) 

>0,/iGR 2 "- 2 

The second factor can be easily evaluated by Lemma 4.4. The first factor itself 
has form (5.1). 

5.2. Separation of variables for the series U(n, n)/U(n) x U(n). Con- 
sider integral (0.5) over a wedge of complex dissipative matrices. Let us repre- 
sent T, S (recall that T — T* , S = -S*) as block ((n - 1) + 1) x ((n - 1) + 1) 
matrices: 

P q* \ s= ( A -b* 

q r J ' \ b ic 
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Repeating considerations of Subsection 5.1, we transform our integral to the 
form 



I dPdAi Y[ 



n-2 j„^[Dl A i-^. 



det[P]^ + 1 dei p\ n -,-2n 

X 



_i detfl+P+Al {<T3_<T3+l||T ^ Tj+l} det(l+P+A)f' T "-ill T "-i} 

P»0,A=-A* XJ ~ 1 1 l 



I u Xn 2n X 



/ 

x [l + u + ic + qP^q* -(q + tyil + P + A)- 1 ^* -b*)Y °"" ^ dudbdqdc 



M>0,6GC"- 1 ,(3GC™- 1 xeR 

{-<r n ||-r n } 



The expression in curly brackets has the form 



|l + u + ic+( q b )x(^ £ ^| 



where the matrix X is the same as above (5.5). Let us change the variable c to 
s by the formula 



1 



s = c--( q b)(X*-X) b 



q 



The Jacobian of the substitution is 1. The expression in the curly brackets 
transforms to 

|l + u + i s +i( q b)(X + X*)( £ )} 
Consider the substitution 

h = &ct{\{X + X*)) 1 ' 2 

Its Jacobian is detP|det(l + P + A)\ 2 (see. Lemma 5.1). Then our integral 
decomposes to a product of the integrals 



f yr dct[P]^ Aj + 1 det pA„_ 1 -2»+l 

J detn+P+Al {CTj_<J3+ll|Tj ~ Tj+l} ' det(HP+A){ CT "-i- 1 H T "-i- 1 }' 



/ (/'' :;; ! 1 + (/ - /.s- -f |/il 2 S- { <T "" ^ dudhdc 



J u x ~- 2n {l + u + is+ \h\ 2 } { 



It remains to calculate multiplier (5.9). 
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5.3. Calculation of separated factor. Passing to spherical coordinates 
in the variable h £ C 2ll ~ 2 ~ R 4 "~ 4 , we transform (5.9) to the form 

2_2n-2 p 

— / u x "- 2n (l + u + ic + p 2 )- a "(l + u-ic + p 2 )- T "p in - 5 dpdudc = 

r(2n-2j J 

ti>0,p>0,cGR 
_2n~2 p 

= — / u x "~ 2n (l + u + ic + v)- an (l + u-ic + v)- T "v 2n ~ 3 dvdudc 

r(2ra-2) J 

u>o,D>o,cet 

Applying Lemma 4.2 we obtain 

dw I dc{(l + w + icy Tn (l + w - icy Tn w Xn ~ 2 } 

(5.10) 



oo oo 

r 2 "- 2 r(A„-2n+l) 



r(A„ - 1) 



For integration in the variable c, we use the formula (see. ]l8| |, 2.2.5.33) 

(Yl 

dx _o_/„ , ^l-U-u V ^ + V - X ) 



2ir(a + by 



{a + ixy(b-ix) v y ' T(p)Y(v) 

— oo 

(where a > 0, b > 0) and obtain 



2vr 2n - 1 r(A rl - 2n + l) 



1 /(2 + » w^dw ■ ^ +1" ~ 1} 



r(A„ - 1) y v r(a„)r(r„) 

o 

Last integral reduces to B-function in the standard way, and finally we obtain 

2 2- ff „-T„ 7r 2n-l r ( An _ 2n + l)r((T n + 7» - A n ) 

T(a n )T(T n ) 

5.4. The case G/if = Sp(n, n)/Sp(n) x Sp(n). Consider integral (0.6) over 
the wedge of dissipative quaternionic matrices B = T + S. Recall that T = T* . 
S = —S*. As above, we represent matrices T, 5* in the form 

T 

where c = cii + C2J + c^k is a purely imaginary quaternion. Repeating consid- 
erations of Subsection 5.2, we reduce integral (0.6) to the form 







s = 


(A -b* \ 
I 6 c 











1 



P>0,A=-A* 3 
X 



'fr 2 det[T]J'~ A ' +1 detP A "-^ 4 "+ 2 
i \ &et[l+P + A) a /-° i+1 det(l + P + A)--i- 4 



J u A "" 4n |l + u + s+|/i| 2 | ° n dudhdc (5.11) 



M>cUieR s "- s ,c=-eeH 
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It remains to evaluate separated factor (5.11). 

Passing to spherical coordinates in variable h e M 8 "~ 8 , we obtain 



27T 4fc - 



T(4fc - 4) 



,A„-4n 



1 + U + p 2 + C 



u>0,p>0,c=-ceH 



^4fc-4 



r(4fc - 4) 



/ 



,A„-4n 



1 + U + V + c 



u>0,v>0,c=-c£H 



T 4fc-4 



r(4fc - 4) 



/ 



,A„-4n 



((1 + m + w) 2 + |c| 2 ) " V 



p 8n - 9 dudpdc = 
v in - 5 dudvdc = 



4n-5 



du dv dc 



M>o,t)>o,c=-eeH 

Then we pass to spherical coordinates in variable c€l 3 , and get 

4tT • 7T 4fe - 4 



r(4fc - 4) 



,A„-4n 



((1 + u + vf + z 2 ) ° n/2 v in -' i z 2 dudvdz 



u>0,v>0,z>0 



2n 



4fe-3 



r(4fc - 4) 



u>0,v>0,x>0 

Integration in x gives 



J u x -- 4n ((l+u + v) 2 + x) an/2 v in ~ b x 1/2 dudvdx 



rM , B(3/2,a„/2-3/2) J u A "- 4n v An - 5 (l + u + vf-^dudv 

u>0,v>0, 

Applying Lemma 4.4, we get the final formula 

27r 4fe - 3 r(3/2)r(a„/2 - 3/2) r(4fc-4)r(A„-4n+l)r(a„-AJ 



T(4fc - 4) T(a n /2) 



T{a n - 3) 



Applying Duplication Formula to r(cr„ — 3), we reduce the last expression to 
the form 

9ff „-4 4fc-3 r ( A « - 4n + 1)I>„ - A„) 



T(a n /2)T(a n /2 - 1) 



§6. Integrals over the Spaces XJ(p,q,K)/XJ(p,K) x XJ(q, K). 



In the Section we evaluate integrals (0.7)-(0.9) over the spaces 0(p, q) /0(p) x 
0(q), U(p, q)/XJ(p) x V(q), Sp(p, q)/Sp(p) x Sp(g). For definitenes, we assume 
P < ?• 
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6.1. The case G/K = 0(p,q)/0(p) x 0(q). Identities (3.5)-(3.6) allow to 
wright integral (0.7) in the form 



dot 



n 



1 L l 
L M 



q-p+j 



L J[ det[l + M + N]° s 



det 



1 L* 
L M 



"(p+<7)/2 



dL dMdN(6A) 



Let us represent M, N as block ((p — 1) + 1) x ((p — 1) + 1) matrices, and L as 
a block ((p — 1) + 1) x (q — p) matrix: 



M - 



P q l 
q r 



N 



A 
b 



-b l 




H 




1 H % 
H P 



l L 



(I 1) 

det(l + M + N) = det(l+P+A) ■ (1 + r - (q+ b){l + P+Ay 1 {q t - b 1 )) 
Denote the expression in curly brackets by u. The integral converts to the form 



/ 



dPdAdH\Y[ 

■3 = 

I 



det 



1 H l 
H P 



Xj -Aj + i 



q-p+j 



det 



1 H* 
H P 



A„-i-(p+g)/2 



V,t X i det[l + P + A]^-^ +1 



dct(l + P + Ay™- 1 



I q ) 



+ [q M (l+P+A)- 1 (l+P+A)- 1 



1 H l 
H P 



q- 
b l 



I' 



(6.2) 



The expression in the curly brackets in (6.2)-(6.3) has form 
1 + u + ( I q b)X 




dudldqdbj (6.3) 



(6.4) 



where 



X 



1 H l 
H P 



+ 





o -s- 1 





-s- 1 



(0 5-) 
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and S denote 

S = 1 + P + A 
As in Subsection 5.1, we wright (6.4) in the form 



! + «+( I Q b)\{X + X*) \ q\ \ \ (6.5) 




Lemma 6.1. 



det ^-(A + X 4 )J =dct y H p ) -det(l + P + Ay 
Proof. det(±(X + X*)) = 



/ / 1 H* \ 1 , / \ / 



= det 



V ( iS' i -^S" ; ) iS 1 - 15'' 1 y 



Then we add the third row to the second row and the third column to the second 
column. We obtain 

( ( 1 Rt V 1 ( ^ \ 

det \ H P J {{l + P-A)-i ) 

V(0 (1 + P + A)- 1 ) \{l + P + A)- 1 + \(l + P - A)- 1 ) 
Formula (1.2) reduces the determinant to the form 
1 iP 



det 



l H H p J •det(i(l + P + yl)- 1 + i(l + P-yl)- 1 - 

"(° d + ^r 1 )^ p )((l + P-A)- 

= det^ ^ ^ dct(l + P + ^)- 1 dct(l + P-^)- 1 x 
xdet(i(l + P-A) + i(l + P + A)-P) ■ 
After the substitution 



( f g b ) Ji(X + X*) =Ag 
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to integral (6.2)-(6.3), we get 
H P 



/n 



dct 



det 



Q-P+3 



1 H l 
H P 



-(p+g)/2+l/2 



^ det[l + P + Ar 



-dPdAdH x 



/ 



/ it 

u>0, hGR'+P- 2 



det(l + P + A)^-!- 1 

A P -(p+ 9 )/2| 1+u+ | ft |2|-P du ^ 



The first factor itself has form (6.1), and the second factor can be easily evalu- 
ated by Lemma 4.4. 

6.2. The cases G/K = \J(p,q)/U(p) x U(q) and Sp(n, n)/Sp(n) x Sp(n). 
A separation of variables in these cases is the same as in the case G/K = 
0(p,q)/0(p) x 0(q). Separated factors coincide with integrals (5.9), (5.11) 
evaluated above. 



§7. Integrals over the Spaces Sp(2n, R)/U(n) 

and Sp(2ra,C)/Sp(n). 



7.1. Integrals over Sp(2n,R)/U(n). We realize this space as in Subsection 
2.4. Let R be a complex symmetric matrix, let R = T + iS, where T, S are 
symmetric. Consider integral (0.10). Let us represent matrices T, S in the form 

Then 

det T = det P ■ (r — qP^q 1 ) 
det{l+T+iS) = dct(l + P+iA){l +r + ic-{q + ib)(l + P + iA)- 1 {q t + ib 1 )} 

Substituting 

u = r - qP^q 1 

we convert (0.10) to the form 

/^(n det|P1 ''" i<+ ' <*J*--<-»> 



_i det[l + p + iA] {aj ~ a3+inTi ~ Tj+l} det(l + P + iA)^-i\\T n -i} 
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where 

/ P" 1 - (1 + P + iA)- 1 -Hi + p + iA)- 1 

V -*(* + P + (1 + P + iA)- 1 

Then we repeat the arguments of Subsections 5.1-5.2. We consider the substi- 
tution 



s =c + Im ( q b ) X (^ q b , 



h = ( q b ) VRcX 
Repeating proof of Lemma 5.1, we obtain 

det(ReX) = det P" 1 |det(l + P + iA\ 
Hence the original integral equals to 



/it 
n 

.7=1 



det[P]; 



•3 A 3 + l 



det[l + P + iA ^i-^+i\\rj-r j+1 } 

A„-i-(n+l)+l/2 



dct[Pr r 

3 dPdAx 



detll + P + a]^- 1 " 1 / 211 ^- 1 - 1 ^ 

x J u x "- 2n {l + u + is+ \h\ 2 } { ~° A ~ Tn} dudsdh (7.1) 

Separated factor (7.1) can be easily evaluated by Lemma 4.4. 

7.2. Spaces Sp(2n, C)/Sp(n). Consider the realization of this space de- 
scribed in Subsection 2.5. Represent the matrices T, S in the form 

^ q *^j S = where q, b E C"; c e C, r E R 

Repeating our calculations, we reduce integral (0.11) to the form 



det[l + P + A)Y/-°° +1 det(l + P + 

-< J u x "-^ 2n+1 '> l + u + cj + *(g,6) ""dudpdbdc^j (7.2) 



where 

6) = g P"Y - (« + 6j)(l + P + Ai)-\q* + j6*) 
Notice that the expression 
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is a quaternion having the form aj + /3k. Hence we can change the variable 
C J = ( c i + c 2i)j to the variable yij + y 2 k by the formula 

yij + y 2 k = cj + §(* (g, 6) - i" 1 ^, 6)i) 

in integral (7.2). Then the expression under symbol of module in (7.2) changes 
to 



1 + u + yd + y 2 k + 6) + i" 1 ^, 6)i) = 

= 1 + u + yj + y 2 k + gP~ Y + 
+ + 6j)(l + P + 4j)"V + + (3 - 6J)(1 + ^ - ATV - 

(7.3) 

The expression in curly brackets has purely complex values (in spite of the fact 
that the expression itself contains quaternionic unit j), moreover this expression 
is hermitian form depending of complex vector (q,b). The determinant of this 
form equals 

detp- 1 |det(l + P + ^j)f 2 
Thus our integral decomposes to the product 



/n 



det[P] 



det pA„-i-(2n-l)+l 



;dPdA X 



^ det[l + P + Aj]?- a ' +1 det(l + P + Aj)^-i-2 

x J u x ~- ( < 2n+1 )\l + u + y 1 3 + y 2 k+\h\ 2 \- an dudy 1 dy 2 dh (7.4) 

where ft- G C 2 ™~ 2 . Factor (7.3) can be easily evaluated in the same way as 
integral (5.11), only integration in c e I 3 is replaced by integration in y e R 2 ). 



§8 Integrals over 0(2n, C)/0(n), SO*(2ra)/U(n). 



8.1. The case G/if = 0(2n, C)/0(2n). Denote by M(R) the algebra of 
real 2x2 matrices. Consider the following elements in M(K) 



x -1 0/ ' VI 0; 

Obviously, 

J 2 = -l, 6» 2 = 1; J6 = -9J (8.1) 
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To any z € C we assign 



z° = a + (3 J = y G M(R) (8.2) 

Matrices z° form a subalgebra in M(R), which is isomorphic to the algebra C. 
Let us denote this subalgebra by C°. Note that the imaginary unite i € C 
corresponds to the matrix J € M(R). Let us define in M(R) also the subalgebra 
R° C C° consisting of all matrices of the form a ■ 1, where a € R. It is easy to 
see that the algebra M(R) is generated by the subalgebra C° and the element 
9. For each z & C we have 

9z° = (z)°e (8.3) 
Define the involution ^ i— » \D r * in M(R) by the formula 

= J^J' 1 

Equivalently for all z, w e C 

(z° + w°6l)* = z° - w°9 = z° - 8w° 

Then (pq)* = q*p*. We stress that the set of fixed elements of the involution * 
coincides with the subalgebra R° . 

Let A be an element of the algebra of k x k matrices over M (R). Let be 
the corresponding 2k x 2k matrix over R. We define a determinant DET(A) as 
square root from | det(Au)|. For matrices over subalgebra C°, this determinant 
coincides with module of the usual determinant of a complex matrix. 

Consider the model of the space 0(2n, C)/0(2n) defined in Subsection 2.6 
(we preserve all notations of Subsection 2.6) . Represent a matrix R in the form 
R = T + H, where R = R\H = -HK Then we wright H as H = SO. Thus we 
represent R in the form 

R = T + S6 where T = T l > 0, S f = -9S6, JT = TJ, JS = SJ 

The matrix R has the size 2n x 2n. We can consider R as n x n matrix 
consisting of 2 x 2 matrices. In other words, we can consider R as matrix over 
M(R). The condition 

R* = J^RJ (8.4) 

(see (2.6)) on this language means that R is *-hermitian (i.e. R as matrix 
over M(R) coincides with matrix obtained by transposition and element-wise 
involution *). 

The conditions JT = TJ, JS = SJ implies that 2x2 blocks of T and S have 
form (8.1), i.e. they are elements of C°. Hence we can consider T , S as n x n 
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matrices over C°. Moreover, the matrix T is a hcrmitian matrix over C°, and 
5* is an antisymmetric matrix over C°. 

In these notations integral (0.14) has the form 



/' n DETlTl Ai ~ Aj+1 
[7 — DET T-^-^dTdS 
t}, DETfl + T + S0]V 3+1 



| ^ J ^3 

J= \DET[1 + T + S6] a / 
Then we wright C°-matrices T, S in the form 



c) T). — — • 

where symbols °*, ot emphasis that we consider transposition and conjugation 
over C°. 

Repeating our calculations, we reduce the integral to the form 

/ n ^3 DETLPlt 3 ~ A3+1 DET pA„-i-(2n-l) 

dP dA 3 . x 

v/ = i DET[1 + P + A0\y-" i+1 DET(1 + P + A6)^-i 



J u A„-(2n-l) DET (! + u + ^ b))-^dudqdb 



where 

*(<Z, b) = qP^q * -(q + W)(l + P + Ae)- 1 {q°* - b ot 9)) 
We wright the last expression in two following ways 

b) = qP^q* -{q + 60) (1 + P + AS)~ 1 {q + b0)* (8.5) 

*(«,&) = (g b)x(f l) (8.6) 



X = 



where 

'P- 1 - (1 + P + AO)- 1 -(1 + P + A9)-H 

eii + p + Ae)- 1 e{\ + p + Ae)- 1 ^ 



It is clear from (8.5), that ^(q, b) € R° (since P and (1+P+.A0) are *-hermitian; 
this implies that ^(q, b) e M(R) is a stable element of involution * ). 

Remark. The summand qP~ 1 q* is not linear over M(R). Hence the expres- 
sion ^(q, b) is not *-hermitian form on module M(R)™" 1 over algebra M(M). 

We have 

*(«,&) = £(*(<?,&) + J-^(q,b)J) 
Applying this equality to (8.6), we obtain 



b°* 
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where 



Y 



P- 1 -±{1+P+A8)- 1 -±(1+P-A8)- 1 -i(l+P+A0)- 1 e+i(l+P-A6»)- 1 e 
±6{l + P+A0)- 1 -±9(l+P-Ae)- 1 ±8(l+P+A8)- 1 0+±0{l+P-Ae)- 1 e 



First notice that the matrix Y doesn't depend on 6, in spite of expression for Y 
contains 9. Formally speaking Y is a matrix over C°. Secondly the matrix Y is 
★-hermitian (since X, J~ X XJ are ★-hermitian) . Hence Y is a hermitian matrix 
over C°. It is easy to show that its determinant equals 

DET(P)- 1 DET(1 + P + AO)- 2 

We obtain 

Vr 2 DET(P]^~ A ' +1 DET pA n _ 1 -(2n-l)+l r 

I L ' J / dPdA x 

f = [ DET[1 + P + Ae]?-"" 1 DET(1 + P + A6)*»-i-* J 



x 

tt>o,/»ec 2 



J u x ~- {2n - 1) W r T{l + u+\h\ 2 )- a ~dudh (8.7) 



Factor (8.7) can be easily evaluated by Lemma 4.4. 

8.2. The case SO*(2n)/U(n). Denote by M(C) the algebra of complex 
2x2 matrices over C. The algebra M(C) contains the subalgebra H°, which 
consists of matrices having the form 



a (3 
a 



(8.8) 



Obviously, the subalgebra H° is isomorphic to the algebra of quaternionsH. 

For a k x k matrix X over M(C), we define the determinant DET(X) as 
square root from determinant of the same matrix considered as 2k x 2k matrix 
over C (we will use this definition only for dissipative matrices, hence square 
root is well-defined). 

We preserve notations of Subsection 2.6. Let us represent the matrix R 
(having the size 2n x 2n and satisfying the conditions R* = J~ 1 RJ) as a sum 
R = T + iS, where T = T*,S = S*\ Then 

T = J- 17 TJ S = J^SJ (8.9) 

As in preceding Subsection, we can consider the matrix R as n x n matrix 
over M(C). Conditions (8.9) mean that T, S are matrices over H°. 
Let us wright integral (0.15) in the form 

/f[ DETP y J+1 „ r DET T-« 2n -VdTdS 

J fJi DET[1 + T + iS]p-^ +1 ^- T '+ l} 
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where i means multiplication by the scalar i in algebra of complex matrices 
having the size 2n x 2n. Then we wright the matrices T, S over H° in the form 

T =( P q °*} S =(t b °*) . where M e H °™ _1 r > c e 
We get 



q r I \b c 



J dPdA^J[ 



DET[T] Aj Aj + 1 DETT A„_i-2(2n-l) 



J[ DET[1 + T + zyl] {<Tj_ ' T3+1 " Tj ' _Tj ' +l} DET(1 + T + i,4) 
x y > M A "- 2 ( 2,l - 1 )DET(l + u + ic+4'( g ,6))^ ff "ll- T ">dwdcd6 ( i^ (8.10) 

where 

*(6, g) = gP" 1 ? * - (« + »6)(1 + P + a)" 1 ^ * + ^6 *) 
This expression satisfies to the identity 

*(M)* = J~ 1 ^(b,q)J 



Hence \&(&, (?) has the form I ^ ^ ] , where /ieC. 



where the symbol * means transposition in the algebra M(C) of 2 x 2-matrices. 

^° ^ 

Consider the substitution 

y = c + lm(q + ib)(l + P + iA)- 1 {q°* + ib *) 

(the symbol Im here means an imaginary part of a complex 2x2 matrix) . Then 
multiplier (8.10) converts to the form 



/ 



u ■ 1) DET(1 + u + iy + Rc^{q,b)) { - a ^- Tn} dudcdbdq 



A„-2(2n- 



The expression Re^(g, b) is a hermitian form over H°, and its determinant is 

DET P~ 1 \ DET(1 + P + iA)\- 2 
Hence the integral transforms to the form 

- DET[P 1 



/ dPdAi rr l —j n T 

J \,- = i DET[1 + P + iA }p-°i+i\\^-T j+1 } 



3 

DET p^-i-2(2n-l)+2 

X DET(1 + P + iA){^-^- 2 \\r n -i-2} x 

X J u A„-2(2n-l) DET '(l + u + ic+ |/ l |2){-<r„H-r„} dudcd6dg ^ 

tt>0,/iGH 2 "- 2 

(8.11) 
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Factor (8.11) can be easily evaluated by Lemma 4.4. 



§9. Plancherel Formula for Kernel Representations. 



Here we discuss only the case of groups G = 0(p,q), considerations for all 
other series are the same. 

For the spaces G/K = 0(p,q)/0(p) x O(q) we consider 'section of wedge' 
model SWp :q (R), see 3.1. We realize a point of the space G/K as a ((q — p) + 
p) x ({q — p) + p) matrix R having the form 

R = Q L M ° + j^j , where M = M*,N — -N*, M — LL l > 

Let s\, . . . , s p be real numbers. Consider the function u s on SW Pt q(R) given by 
the formula 

u s (R) = JJdet[M - LL t}-Vz+i(»i-si+i) 

Recall that (see 3.2) that these functions are eigenfunctions of a minimal parabolic 
subgroup y C G. Denote by tp s the average of the function u s under the action 
of the maximal compact subgroup K = 0(p) x 0(q) C 0(p,q): 



(p a (R) = / u s (RW)dk 



K 

where dk denote Haar measure on K (normalized by the condition: the total 
measure of the group is 1) and R^ is a fractional linear transformation defined 
by the formula (2.3). 

It is easy to see that <p s are exactly spherical functions on the symmetric 
space G/K(see for instance ||, 

Berezin function Ti a in our coordinates is given by the formula 

4«(p+9) det(M -LL l ) a 
" ( ' ~ det(l + M + N) 2a 

(see Subsection 3.3). Its spherical transform (see @,@) equals to 



A a (s)= H a (R)(p- s (R)dL dM dN 

J G/K 

By X-invariance of the function H a (R) , this integral equals to the integral 

T> a (R)u- s (R)dL dMdN 

G/K 
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The last integral is a partial case of integral (0.7). Thus 

= const A I> + is, - (g +p)/4 + l/2)r(a - ^ - (g +p)/4 + 1/2) 

By Gindikin-Karpelevich inversion formula (see. M, M) 



~B a (R) = const / 

J si 



A Q ( s )^ s (i?)ci/x(s) (9.1) 

>S 2 >...>S„>Q 



where dfj,(s) is Gindikin-Karpelevich measure. 

Wrighting (9.1) in the explicit form we obtain the following theorem 

THEOREM 9.1. Let a > (q + p)/4 - 1/2. Then 

^'S ^-l-')) ' 

'' r(a + is k - (q + p)/4 + l/2)T((q - p)/2 + is k ) 



>< n 



r(l/2 + i(a fc - aj))r(l/2 + i(s k + s,)) 2 



r(i(sk - s!))r(«(s fc + s;)) 

Proof. Formula (9.2) is certainly correct if 

a {R) < oo, / |^4 Q (s)| d/i(s) < oo 



r(zs fc ) 

ip s {R)ds\ds2 ■ ■ -ds p (9.2) 



(see 0, Theorem IV.9.5., [|, Theorem XIV.5.3). The both conditions are satis- 
fied for sufficiently large a. By analytical continuation considerations, formula 
(9.2) is correct for all a > (q +p)/4 - 1/2. ■ 
Remark. At the point a = (q + p)/4 — 1/2 in the right part of (9.2) there 
appears a pole in numerator. At the same point 23 a became not square integrable 
on the group G — 0(p,q). As it is shown in 0, for a < (q + p)/4 — 1/2 the 
spectrum of a kernel representation is not contained in principle series. By this 
reason, Theorem 9.1 for a < (q+p)/4 — 1/2 is not correct. 
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